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Abstract 



This study focused on defining and delineating core skills and tasks needed for 
successful CTE student educational growth and success through a curriculum 
and assessment alignment process. The context for this project lies within 
Automotive Service Technology (AST), which must additionally meet the National 
Institute of Automotive Service Excellence (ASE) program certification 
standards. Unfortunately, two skill areas are not clearly defined or assessed by 
ASE: Basic Vehicle Interval Maintenance Skills and Basic Vehicle Repair Skills, 
which combine to form the Automotive Service Technology Foundational Skills 
(ASTFS). A qualitative analysis of emergent systems patterns was used to 
construct a table of specifications for ASTFS, which can facilitate curriculum 
development and assessment. Specifically, this process defined and delineated 
prerequisite technical skills and their associated ability domains of knowledge, 
concepts, and skills so they could be taught, learned, and assessed. 


Criterion Referenced Assessment: Delineating Curricular 
Related Performance Skills Necessary for the Development 
OF A Table of Test Specifications 

Peter D. Hart Research Associates reports that 40% to 45% of high school graduates are 
being under-prepared for academics, work habits, as well as for job specific skills required for 
college or the workforce (2005). The United States Department of Education‘s Strategic Plan for 
2007 to 2012 outlines focused initiatives for educational reform (2007). The third goal of the 
strategic plan centers on students ‘ successful transitioning between next stage levels: secondary 
education, post secondary education, and the workforce. Mandated indicators of success have 
been mandated and schools are expected to improve student's skill levels to prepare the student 
for their next stage. 

Ensuring high levels of student success transitioning from one stage to the next requires 
alignment of learning materials, learning plans, learner practice activities, and learner assessments 
with prescribed educational standards (Solution Tree, 2006). Educational accountability is 
further mandated by the re-authorization of the Elementary and Secondary Education Act (ESEA) 
of 2001 (more commonly known as the No Child Eeft Behind (NCEB) Act) and is a co mm on 
theme that affects everybody (i.e. staff, teachers, and students) in every school and throughout 
America. NCEB focuses on accountability improvement of graduates ‘ skill levels for either post 
secondary education or the workforce. Students who possess higher academic and technical skills 
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have more options and a higher probability for success (United States Department of Education, 
2003). 


Career and Technical Education (CTE) is a nationwide program that emphasizes training 
for primary, secondary, and post secondary educational stages for the workforce needs of today 
and tomorrow's society. Although there are many parallels between CTE programs and liberal 
arts college prep curricula, there are also important differences. The most important difference 
between college prep and CTE program curricula is the learner outcome. Specifically, many CTE 
programs focus on academic integration of specific skill standards and tasks and are co mm on 
language in CTE curricula as much as academic benchmarks are in college prep curricula. Skill 
standards are typically up-to-date task listings, but can also include program specifications for 
curriculum, learning materials, the learning environment. In either case, whether addressing 
standards-based benchmark skills or skill standards in CTE, assumptions are made frequently and 
possibly inappropriately, regarding prerequisite learner level of content and skills upon entry into 
the curriculum (Diamond, 1998; Eezotte, 1992; Solution Tree, 2006). One problem that is 
confounding CTE teachers and learners today is that basic mechanical skills learned in traditional 
high school programs have disappeared along with those basic programs (Grub & Eazerson, 
2004). If prerequisite underlying skills are ignored, assumed, or minimized, teachers‘ will find it 
difficult for all students in a class to reach desired levels of achievement (Eezotte, 1992). CTE 
learner outcomes encompass skill sets that often include assessment of skills such as mechanical 
aptitude, specific job level content, problem solving skills, as well as task achievement. 
Assessment results for a given skill set can be used to determine students‘ level of preparedness. 

Identifying and assessing learner levels for prerequisite content and skills, instead of 
assuming those levels, allows tailoring of the learning process to individuals and is key to 
instructional sequencing and planning activities, which help teachers and learners eapitalize on 
their educational growth potential (Diamond, 1998; Eezotte, 1992; Solution Tree, 2006). 
Identifying and defining prerequisite and underlying skills or tasks requires purposeful analytic 
techniques (Jonassen,Tessmer, & Hannum, 1999). Automotive Service Technology (AST) tasks 
are hierarchical in nature and naturally underlie other lower level or predicate tasks that make up 
a higher level or complex task (Jonassen et ah, 1999). For example, if the task is to ask Susan or 
Johnny to, "Replace the window wiper blade inserts on the family vehicle,” this assumes several 
underlying prerequisite skills or tasks. One underlying task involves knowing the proper tools 
and parts to use for the job task. Another underlying task would involve knowing or locating the 
exact procedures and tools to use to replace the wiper inserts. Some other underlying tasks may 
involve an understanding of when to replace the wiper blades instead of the wiper inserts. Still 
another lower level set of skills or tasks can be identified such as listening skills, ability to 
understand locations, and ability to physically coordinate the task steps. 


Purpose of the Study 

The purpose of this study was to define and delineate skills and tasks relevant to the 
ASTFS that enable curriculum and assessment development of the ASTFS. Proper assessment 
design and construction requires that specific content and abilities are very well delineated. To 
achieve the purpose for this study, objectives that align with typical initial assessment design and 
eonstruction processes are used. 

1. The assessment design and construction processes utilized an assessment purpose and 
methods to ensure both the content and ability domains were proportionately aligned with a 
highly recognized eontent or skill area. 
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2. The content and ability domains for the ASTFS are delineated and communicated well 

enough to enable teaching, learning, and assessment of the ASTFS. 

Automotive Service Technology (AST) will provide the contextual basis for the processes 
described in this paper. A second paper following this writing will describe the processes used to 
make a Criterion Referenced Test using the ASTFS as a context. The purpose of the intended 
ASTFS Proficiency (ASTFSP) Assessment is to provide a current or prospective AT employee 
with their proficiency level of the ASTFS as compared to industry criterion levels. 

To achieve the objectives of this assessment study several processes were implemented 
that involve the alignment of skills, curriculum, and assessment. First, a Learning Flierarchical 
Task Analysis (LFITA) process, as essentially modified from Jonassen, Tessmer, and Flannum 
(1999), was used to define and delineate the prerequisite skills and tasks from a set of job level 
tasks. Second, an Analysis Matrix Summary Table (AMST) process, as essentially modified 
from BarteTs Task Selection Chart Procedure (1976) was used to prioritize and proportion the 
skills. Third, a Task Domain Categorization Analysis (TDCA) was used to represent the ability 
domain of knowledge, concepts, and skills to define curricular related teaching and learning 
objectives. Finally, a Table of Test Specifications was used to represent a balanced assessment 
plan that could be used for formative or summative proficiency assessments. 


Methods 

The first steps in the assessment design process will be to fulfill the first objective, 
identify the purpose/s of the assessment and align the content ability domain proportions. 
Assessments can be made to evaluate the progress and proficiency of a group, person, or student 
in either formative or summative form. Moreover, assessment can be designed to discriminate 
and separate proficient students from non-proficient students in nearly any employable or 
educational area regardless of the context. The academic areas that are parallel to CTE are the 
state high school assessments required by the NCLB Act. These serve as discriminating 
standards of educational preparedness as well as indicators of success for any given school. 
Fligh-quality instructional remediation is dependant on well defined skills as well as high-quality 
formative assessments to evaluate students ‘, teachers, class, or program‘s performance. 

The key to effective assessment of a skill area is the accurate defining and delineation of 
the underlying, assumed, or prerequisite tasks or skills within the particular educational skill area. 
Knowledge and skills change with society, time, and technological advances. One example of a 
training area that is affected by society, time, and technological advances would be in automotive 
repair (United States Department of Labor, 2006). The quantity and rate of technological 
advance over the past several decades requires that a do-it-yourself person possess higher skill 
levels than the average American car owner has. The implications of society, time, and 
technological advancement indicate that prerequisite skills and tasks need to be redefined and 
delineated periodically for accurate alignment of curricula and assessments for employment or 
post secondary education preparation in a particular CTE area. 


The Context of a Skill Area 

The National Institute of Automotive Service Excellence (ASE, 2005) is a nationally 
recognized organization that assesses and certifies Automotive Service Technology (AST) 
technicians in eight vehicle system areas: Engine Repair, Automatic Transmission and Transaxle, 
Manual Drive Train and Axles, Steering and Suspension, Brakes, Electrical and Electronics, 
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Engine Performance, and Air Conditioning. Individually an AST technician can take as many 
certifications as desired. Surprisingly, not all automotive technicians are required to hold ASE 
certifications. In fact, many of the individuals employed in "Oil Change" shops may not possess 
any certification. Instead of certification, "hands-on" On the Job Training (OJT) is relied upon 
and passed on to the consumer. 

In addition to the eight AST systems there are two sets of skills that are not directly 
assessed by ASE certification assessments. One skill set is core to all eight AST system areas: 
Basic Vehicle Interval Maintenance Skills (BVIMS). The second skill set corresponds to the 
underlying skills needed to perform basic vehicle system level diagnostic and repair tasks (eg. 
Identify and use hand tools correctly) and are collectively referred to as the Basic Vehicle Repair 
Skills (BVRS). The combination of the BVIMS and BVRS skills together form the set of 
Automotive Service Technology Foundational Skills (ASTFS). 

ASE certified programs are evaluated through a multi-site expert group process and have 
aligned their AST programs with the standards mandated by the National Automotive Technician 
Education Foundation (NATEF) (2005). NATEF provides 10 standards that guide the program 
curriculum, laboratory facilities, program management, and tool and equipment requirements 
necessary for students to learn the documented list of job level tasks that are called for in an AST 
curriculum. 

The NATEF tasks are nested within one of the eight possible related automotive system 
areas, which parallel the eight ASE certification test areas. The tasks within any given 
automotive system area are further nested within their related duty or unit areas. For example the 
brakes area has seven duty or unit areas and includes: general brake systems diagnosis, hydraulic 
system diagnosis and repair, drum brake diagnosis and repair, disc brake diagnosis and repair, 
power assist units diagnosis and repair, miscellaneous (wheel bearings, parking brakes, electrical, 
etc.) diagnosis and repair, and lastly antilock brake and traction control systems (NATEF, 2008). 
Surprisingly however, either separately or as a set, the ASE certification exams share a co mm on 
set of "assumed" skills and knowledge that are not explicitly measured by these exams. 
Moreover, NATEF leaves it to the instructors and curriculum developers of CTE programs to 
identify and teach these basic prerequisites. 

Tasks are clusters of skills and activities ranging in complexity from lower level simple 
or underlying tasks to higher level complex tasks (Jonassen et al., 1999). Within the NATEF job 
level tasks are assumed lower level skills and tasks that are often prerequisite components to 
those job level tasks. The assumed prerequisite skills that NATEF directly cites as impacting the 
NATEF system tasks include general skills such as: safety, identification and use of tools and 
equipment, locating and using reference and training materials (reading and locating information 
academics), following manufacturers recommended procedures (contextual reading 
comprehension), storage and use of hazardous materials as outlined by OSHA, and the procedural 
handling of toxic materials in accordance with EPA and various levels of regulatory government 
(NATEF, 2008). 

The NATEF tasks are arranged by vehicle system areas and are up-to-date, clearly 
defined, higher level job tasks. However, the assumed prerequisite skills are generalized lower 
level skills or tasks, which lack clear definition and delineation. Currently a reliable and valid 
singular assessment does not exist to report a student or technician^ proficiency or progress level 
on the ASTFS to allow a proper evaluation of a person‘s preparedness and ability of the ASTFS. 
A singular assessment for these assumed skills is important to complement existing assessments 
available for each automotive system through ASE certification tests. An ASTFS assessment 
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could be used to assess a person‘s proficiency level whether he or she is leaving an AST 
educational program or upon entry to the AST industry field. Additionally, an ASTFS 
assessment could be used for program evaluation purposes regarding effectiveness of the teaching 
and learning environment. 


Skill Domain Discovery Process 

To complete the first objective a survey of methods were initiated to discover the process 
that would serve the delineation process best. The first objective of this study was: 

The assessment design and construction processes utilized an assessment 
purpose and methods to ensure both the content and ability domains 
were proportionately aligned with a highly recognized content or skill 
area. 

Several methods are available that can be used to define component elements of a 
complex task. Develop A Curriculum (DACUM) or a common Delphi process uses content 
experts to identify the job level tasks or elements within a given task (DACUM, 2005). However, 
if the experts are limited in their knowledge or their understanding of the skill or task area, then 
the DACUM or Delphi process may not reveal some of the prerequisite skills or tasks required for 
job level task proficiency. Because there has been a history of automotive skill incompetence 
documented over the years in common newspapers etc. it doesn‘t seem a like it would be a good 
assumption that current Automotive Technicians (AT)‘s or AT teachers possess expertise levels 
of the previously undefined ASTFS. In order to assure that the defined skills are completely 
delineated, an objective analytical approach was sought. Specifically, the desired logical and 
objective process selected for the important and essential assumed skills used an analytic 
qualitative process based on the NATEF supplied tasks by implementing a Learning Hierarchy 
Task Analysis (LHTA) (Jonassen et al., 1999). 

The steps to perform a LHTA listed by Jonassen, Tessmer, & Hannum (1999) include:!) 
become familiar with the subject, 2) state the task, 3) identify the beginning leamer‘s entry level 
of knowledge for the task, 4) identify the first level prerequisites, 5) identify second level 
prerequisites, 6) identify subsequent level prerequisites, 7) determine limit of prerequisite levels 
of discovery desired, 8) construct the hierarchy of levels, 9) verify the hierarchy logically and 
empirically. Refer to Figure 1 for a graphical representation of a Learning Hierarchical Task 
Analysis for an AST NATEF task, which does includes academic content. 

Notice in Figure 1 that the career area, task area, duty area, and job level task are labeled 
in boxes at the top of the diagram. Below the task are eight task steps required to complete the 
NATEF task. These are delineated from left to right in dark shadowed boxes. Below the task 
steps are oval boxes that further delineate hierarchically underlying lower levels of tasks, which 
are linked to task steps. The hierarchical explanation lays in the graphical depiction of the lower 
oval boxes in reference to the job level task steps, which represents the delineated simpler lower 
level skills and tasks. The actual tool names are generalized in the graphic boxes because they 
depend specifically on the year, make, and model of the vehicle being worked on. The skills 
purposefully excluded in the LHTA for this study were related to academic content, personality 
traits, and attitudinal constructs such as employability work habits. These skills were considered 
too broad to be included within a single criterion referenced assessment. 
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Hierarchically Lower Tasks Task Steps 


Figure 1. Learning Hierarchical Task Analysis Example 

Hierarchical Task Analysis Example 


Career Area: Automotive Service Technology 

Area: Electrical/Electronic Systems 

Duty: Horn and WiperA/Vasher Diagnosis and Repair 


Read 

complaint on 
Repair Order 


Task: Diagnose incorrect wiper 
operation; diagnose wiper speed control and 
park problems; perform necessary action. 


I 

Verify 
customer 
complaint and 
decide 
diagnostic 
steps 


Request parts 
cost, parts 
availability, and 
estimate of time 
for corrective 
actions 
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Learning Hierarchical Tasks Analysis Procedures 

The LHTA referenced almost 400 NATEF tasks for the most common six of the eight 
system areas. The two areas omitted for this first portion of the LHTA include: Automatic 
Transmission and Transaxle and Manual Drive Train and Axles. These areas are typically 
specialized and are dependant on skills from several of the other six NATEF areas. Several 
strategies were used to make the LHTA process manageable. First, the actual analysis process 
was completed using a relational database program, which directly referenced the six NATEF 
task areas. Second, the prerequisite skills and tasks were gathered and sorted into multiple 
categories for each NATEF task. 

An iterative qualitative process of reviewing and comparing the prerequisite skills for the 
NATEF vehicle system areas, duty areas, and job level tasks allowed several patterns to emerge. 
These formed the predefined categories related to tool and information type for the initiation of 
the LHTA process. The nine emergent LHTA categories include: Hand Tools, Power Tools, 
Special Tools, Shop Equipment, Chemicals, Hazards, Toxic Handling, Safety Procedures, and 
Other Assumed Tasks. One end product of the LHTA process is a flat sheet report with NATEF 
tasks in rows by the nine descriptive categories in columns. 

Although specific tools and equipment would not be listed in the analysis, it was still 
necessary to have a defined list compiled during the analyzing process for accurate referencing 
purposes. Specific tools and equipment were further sorted into various logical sub-types. For 
example, the hand tools category encompassed nine sub-types and included: general fastener 
tools, non-threaded fastener tools, general electrical service tools, impact fastener tools, pullers, 
cleaning tools, cutting tools, threading tools, and last measuring tools. Therefore, the hand tool, 
power tool, and special tool categories were further defined by listing tools within smaller sub- 
type categories. AST tool catalogues and AST related text books were researched and referenced 
for specific information concerning non-vehicle manufacturer specific tools and equipment. 

Qualitative data in the form of descriptive information and references to tool or 
equipment sub-type categories that were appropriate for each of the nine categories were recorded 
within the relational data base form for each of the nearly 400 NATEF tasks. The skills and tasks 
as a group made up the BVRS area of the ASTFS. An analytical extraction process was then 
used to re-sort the redundant qualitative categorical data into either new or existing categories and 
sub-categories that are more homogenous to the types of skill content instead of the type of tool 
or information category. Table 1 presents a crosswalk of the new category types in relation to the 
original nine categories. 

Table 1 is read as follows: the first of the three sub-scales of tasks for the BVRS scale in 
the first column, "Oxy-Acetylene Torch Safe Usage.” The first task in that particular sub- 
category is listed in the next lower row of the first column as, "Oxy-Acetylene Torch Set-up.” 
Two X‘s were recorded under each of two of the nine categories listed in columns 5 and 9 
indicating that this prerequisite task relates to shop equipment and safety procedural information. 
Additionally, column 10 is the category, "Other Assumed Tasks,” and includes a worded 
description for this particular ASTFS task concerning relations to torch usage and safety. Thi s 
indicates additional and important knowledge and practical skills concerning torch usage that 
should be specifically included for this task. The balance of the table can be read in a similar 
manner. 

Table 1, 

Basic Vehicle Repair Skills Descriptors and Categorization Scheme by Learning Hierarchical 
Task Analysis Categories 
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Basic Vehicle 
Repair Skills sub- 
scale categories 

Hand 

Tools 

Power 

Tools 

Special 

Tools 

Shop 

Equipment 

Chemicals 

Hazards 

Toxic 

Handling 

Safety 

Procedures 

Other 

Assumed 

Tasks 

Oxy-Acetylene 
Torch Safe 
Usage Scale 










Oxy-Acetylene 
Torch Set-up 




X 




X 

Torch use 
and Safety 

Oxy-Acetylene 
Torch Storage 




X 




X 

Torch use 
and Safety 

Oxy-Acetylene 
Torch Practices 




X 




X 

Torch use 
and Safety 

Mechanical 
Aptitude & Safe 
Tool Use Scale 










Pneumatic tools 
and equipment 


X 


X 




X 


Electrical tools 
and equipment 


X 


X 




X 


Hand tool 
selection and use 

X 







X 

Threaded 
& non- 
Threaded 

Mechanical 

Aptitude 

X 







X 

Threaded 
& non- 
Threaded 

Facility 

Equipment Use 
and Safety Scale 










Hoists and jack 
use 




X 




X 


Fire extinguisher 
selection and use 




X 


X 


X 


Ventilation 




X 

X 

X 


X 


Personal 

Protective 

Equipment 




X 

X 

X 

X 



Environmental 

Concerns 




X 

X 

X 

X 




During the LHTA re-sorting and crosswalk process, several items (not listed) in the 
— Othe Assumed Tasks” category did not fit into the new categorical schemes. Thus, a second 
ASTFS categorical area emerged for these tasks. Research of various manufacturer ‘s service 
information revealed an existing categorical title for the misfit tasks, which didn‘t align with the 
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BVRS category. These misfit tasks in the "Other Assumed Tasks” category are commonly 
referred to as Interval Maintenance tasks. Therefore, the title for the Interval Maintenance tasks 
became the Basic Vehicle Interval Maintenance Skills (BVIMS). The categorical area skills in 
the BVIMS were further sorted into vehicle maintenance interval mileage service time categories, 
which included: 3,000 to 7,500 mile interval maintenance, 15,000 mile interval maintenance, and 
30,000 mile interval maintenance. The end result of the LHTA process was that the ASTFS tasks 
were sorted into hierarchical categorical scales of skills with the related ASTFS tasks listed under 
each sub-scale category. Although some manufacturers are changing their interval maintenance 
category schemes, the maintenance tasks remain stable. The two scale task lists, BVIMS and 
BVRS, together form the complete ASTFS. 


Empirical Validation Procedures 

The next step of the LHTA was to further validate the ASTFS empirically by plotting the 
ASTFS against all eight of the NATEF areas forming an Analysis Matrix. The Analysis Matrix is 
a mixed methods process used to empirically confirm the integrity, validation, and verification of 
the ASTFS content as outlined by Bartel (1976). This process was also used to ensure the 
alignment of the ASTFS tasks with job analysis information as required by the credentialing 
standards for measurement (Joint Committee on American Educational Research Association, 
American Psychological Association, & National Council on Measurement in Education [Joint 
Committee], 1999). The Analysis Matrix listed the ASTFS tasks in rows with the NATEF duty 
areas nested within the eight NATEF vehicle system areas in columns. A coding system was 
utilized to indicate the relatedness of the ASTFS task as either being an underlying task match, a 
partial task match, an exact task match, or no task match to each of the columns of the NATEF 
duty areas. An underlying task match is one that is assumed by the job level task implicitly and is 
not normally described in the repair procedure steps. A partial task match is one that may be 
described in the procedures and is required to perform the repair procedures and steps. The end 
result of the Analysis Matrix was a summary table, which contained a set of categorized 
frequencies for each ASTFS task descriptor, indicating the relatedness and importance of each 
ASTFS task. Refer to Tables 2 and 3 for a listing of the BVRS area and BVIMS area Analysis 
Matrix Summary Tables listing the frequencies for each coded area. 

The Analysis Matrix Summary Table information for Table 2 is read as follows. The first 
of three sub-scales listed in column one is the Oxy- Acetylene Torch Safe Usage sub-scale. The 
first ASTFS task for this sub-category is listed in the row directly below and is labeled Oxy- 
Acetylene Torch Set-up. The second, third, fourth, and fifth column for the ASTFS task row 
indicates, respectively, the count of match types where this ASTFS task has either an assumed 
relatedness, a partial relatedness, an exact relatedness, or no match relatedness concerning the 
skills needed to proficiently complete the NATEF job level tasks within the NATEF duty area. 
The second, third, fourth, and fifth column for the ASTFS task sub-category row indicates, 
respectively, the percent of match types among the ASTFS tasks where this ASTFS task has 
either an underlying task match, a partial task match, an exact task match, or no task match 
concerning skill relatedness needed to proficiently complete the NATEF job level tasks within the 
NATEF duty area. The sixth column for each ASTFS task row indicates the total percentage of 
all four match types for all 45 duty areas of the eight NATEF areas. The seventh and eighth 
columns for each sub-category scale row indicate the percentage of matches for each scale and 
for the sum of the sub-scales together, respectively. The balance of the ASTFS tasks and sub- 
scales listed in Tables 2 and 3 can be read in the same manner. 

In viewing the frequency results it can be noted that the BVRS tasks are mostly an 
underlying task match in relation to the NATEF tasks and the BVIMS tasks are a partial task 
match in relation to the NATEF tasks. The percentage of each sub-scale indicates weighting of 
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importance of each ASTFS category and sub-category to the NATEF duty areas as a whole. The 
percentage weighting can serve as a general guide to the instructional process and plan as well as 
for assessment guidance and should not be reflective of strict adherence to content. 
Consideration should also be given to the breadth of complexity within each category. For 
example the Facility Equipment Use and Safety sub-scale (54.80%) outweighs the Mechanical 
Aptitude and Tools Use sub-scale (36.87%) in the Analysis Matrix Summary Table percentage 
results because nearly every repair requires facility use, but not necessarily mechanical aptitude 
or tool use. Flowever, consider that there are hundreds of tools required in the repair process for 
the Mechanical Aptitude and Tools Use sub-scale category scale in comparison to the Facility 
Equipment Use and Safety sub-scale category. The difference in percentage indicates that 
Facility Equipment Use and Safety sub-scale category is more related (important) to the NATEF 
job level tasks (54.80%) and is not necessarily a larger or more difficult area of content and skill 
learning than the other areas. The next step in the process is to further define the ASTFS tasks to 
reflect recent changes in technology and is expected to change more frequently in time. 


Table 2 


\Analysis Matrix for the 2005 NATEF Areas and the ASTFS Basic Vehicle Repair Skills Scale 


Basic Repair Skill 
Areas 

Sub-Scale Areas 

Assumed 
Task 
Match (*) 

Partial 

Task 

Match 

(P) 

Exact 

Task 

Match 

(X) 

No 

Task 

Match 

%of 

Area 

Match 

Percent 
of Scale 

Oxy-Acetylene 
Torch Safe Usage 
Sub-Scale 

Oxy-Acetylene Torch 
Set-up 

11 

0 

0 


24.4% 


Oxy-Acetylene Torch 
Storage 

11 

0 

0 


24.4% 


Oxy-Acetylene Torch 
Practices 

11 

0 

0 


24.4% 


Sub-Total 


100.00% 

0.00% 

0.00% 

0.00% 


8.33% 

Mechanical 
Aptitude & Safe 
Tool Use Sub- 
Scale 

Pneumatic tools and 
equipment 

32 

0 

0 


71.1% 


Electrical tools and 
equipment 

38 

0 

0 


84.4% 


Fland tool selection 
and use 

41 

0 

0 


91.1% 


Mechanical Aptitude 

35 

0 

0 


77.8% 


Sub-Total 


100.00% 

0.00% 

0.00% 

0.00% 


36.87% 

Facility 

Equipment Use 
and Safety Sub- 
Scale 

Floists and jack use 

45 

0 

0 


100.0% 


Fire extinguisher 
selection and use 

45 

0 

0 


100.0% 


Ventilation 

41 

0 

0 


91.1% 


Personal Protective 
Equipment 

45 

0 

0 


100.0% 


Environmental 

Concerns 

41 

0 

0 


91.1% 


Sub-Total 


100.00% 

0.00% 

0.00% 

0.00% 


54.80% 

Total 







100.00% 
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Table 3. 


Analysis Matrix for the 2005 NATEF Areas and the ASTFS Basic Vehicle Interval Maintenance 
Skills Scale 


Basic Vehicle 
Interval 
Maintenance 
Skills 

Interval Maintenance Sub- 
Scale Areas 

Totals 

%of 

Area 

Match 

Percent 
of Scale 

Assumed 

Task 

Match 

(*) 

Partial 

Task 

Match 

(P) 

Exact 

Task 

Match 

(X) 

No 

Task 

Match 

3,000 to 
7,500 mile 
maintenance 
Sub-Scale: 

Change oil and filter 

0 

0 

1 


2.2% 


Lube chassis and drive -train 

0 

3 

0 


6.7% 


Check/Service all fluid levels 

0 

7 

0 


15.6% 


Check/Locate Fluid leaks 

0 

7 

0 


15.6% 


Lube vehicle access features 

0 

0 

0 

1 

0.0% 


Check/Service clutch free 
play 

0 

1 

0 


2.2% 


Check/Service drive belts 

0 

3 

1 


8.9% 


Perform Safety Inspection 

0 

11 

0 


24.4% 


Check/Service tire pressure 

0 

0 

1 


2.2% 


Check/Service all hoses 

0 

3 

1 


8.9% 


Check/Service battery and 
cables 

0 

0 

1 


2.2% 


Check/Service MIL light, 
engine, body codes 

2 

0 

0 


4.4% 


Totals 


4.65% 

81.40% 

11.63% 

2.33% 


56.58% 

15,000 mile 
maintenance 
or One year 
maintenance 
Sub-Scale: 

All of the 3,000 mile 
maintenance areas: 

0 

0 

0 




Check/Service tires and 
wheels 

0 

0 

1 


2.2% 


Replace air filter 

0 

1 

0 


2.2% 


Check/Service all hoses 
(Coolant and Vacuum) 

0 

2 

0 


4.4% 


Check/Service cooling 
system & A/F protection 

0 

0 

2 


4.4% 


Clean radiator externally 

0 

1 

0 


2.2% 


Check/Service tires & wheels 
(rotate tires/wheels) 

0 

0 

1 


2.2% 


Check/Service emissions 
filter 

0 

0 

1 


2.2% 


Check/Service brake 
components 

0 

6 

0 


13.3% 


Check/Service steering and 
suspension components 

0 

0 

2 


4.4% 


Check/Service vehicle 
condition (cosmetically) 

0 

0 

0 

1 

0.0% 
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Maintenance the battery (if 
applicable) 

0 

0 

1 


2.2% 


Check/Service C.V. joints 
and suspension 

0 

2 

0 


4.4% 


Lube CV joint boots 

0 

0 

0 

1 

0.0% 


Lube door seals 

0 

0 

0 

1 

0.0% 



Replace spark plugs 
(optional) 

3 

0 

0 


6.7% 



Replace air cabin filter 

0 

0 

1 


2.2% 


Totals 


11.11% 

44.44% 

33.33% 

11.11% 


35.53% 

30,000 mile 
maintenance 
or Two year 
maintenance 
Sub-Scale: 

All of the items of the 1 5 K 
maintenance 

0 

0 

0 




Flush brake fluid 

0 

0 

1 


2.2% 


Flush auto-trans fluid 
(optional) 

0 

3 

0 


6.7% 


Flush cooling system 

0 

0 

1 


2.2% 


Replace fuel filter (optional) 

0 

0 

1 


2.2% 


Totals 


0.00% 

50.00% 

50.00% 

0.00% 


7.89% 

Total 







100.00% 










Skill Definition and Representation 

To complete the second objective several additional processes were needed for further 
delineation for curricular and assessment communication purposes. The second objective was: 

The content and ability domains for the ASTFS are delineated and communicated 

well enough to enable teaching, learning, and assessment of the ASTFS. 

A general listing of the ASTFS tasks and categorical group scale and sub-scales serves as 
a defining and representing guide to the skills domain and are listed in Table 4. General skill 
listings are ambiguous in comparison to further clarified, defined, and specific descriptions of 
lower level skills. Flowever, general skill listings are useful for skill presentations as represented 
for test specifications, area and unit goals, or summary skill explanations. General listings are 
probably most commonly used in test specifications to ensure proper item proportioning and 
evidence of content validity. Under the sub-scales are the general outcomes and general 
objectives used for further defining the skills. 

The next step in the representation and defining process is to further define and delineate 
the ASTFS tasks, specifically with an ability domain. The defining process for each of the 
ASTFS task sub-scales required researching reputable sources concerning safety and proper 
procedures from such agencies and organizations as OSFIA, EPA, the American Welding Society, 
Mitchell-One, and specific manufacturer ‘s vehicle repair information. The research information 
provided guidance for further specific defining of each ASTFS task concerning the categories of 
knowledge, concepts, and skills. The end result of the processes described in the next two 
sections was a content validated outline of categorized unit outcomes with one to three levels of 
specific objectives. The specific objectives have been defined and delineated for purposes of 
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curricular alignment, construction of educational materials and instructional lessons, and 
designing and construction of assessments. 

Table 4, 

List ing of General Categorical Groups, Sub-Scales, and Outcomes for the ASTFS 

010 Automotive Service Technology Foundational Skills 

A. Basic Vehicle Repair Skills 

1 Demonstrate Oxy-Acetylene Torch Use, Safety, and Maintenance 

a) Set-up oxy-acetylene torches safely 

b) Store oxy-acetylene torches 

c) Practice safe oxy-acetylene torch usage 

2 Demonstrate Mechanical Aptitude & Safe Tool Use 

a) Select and use pneumatic tools and equipment safely 

b) Select and use electrical power tools and equipment safely 

c) Select and use hand tools safely 

d) Apply Mechanical Aptitude to solve problems 

3 Demonstrate Safe Facility Equipment Use 

a) Use vehicle hoists and jacks safely 

b) Select and use fire extinguishers safely 

c) Practice ventilation precautions 

d) Understand and practice correct Personal Protective Equipment (PPE) usage 

e) Demonstrate correct procedures for environmental concerns 

B. Basic Vehicle Interval Maintenance Skills 

1 Demonstrate 3,000 to 7,500 mile Interval Maintenance Procedures: 

2 Demonstrate One Year or 15,000 Mile Interval Maintenance Procedures 
(Includes the 3,000 mile maintenance areas): 

3 Demonstrate Two Year or 30,000 Mile Interval Maintenance Procedures 
(includes the 15 K maintenance items): 
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Task Domain Categorization Analysis 

The Task Domain Categorization Analysis (TDCA) process described is a hybrid process 
referencing concepts from the AMST created during the AMST process (Jonassen et ah, 1999) 
and from analytic procedures described for curricular alignment (Stiggins, Arter, Chappuis, & 
Chappuis, 2004; Chapuis & Chapuis, 2002). The purpose of this process is to logically simplify 
the curricular aligning and defining process to enable further definition and clarity of the skills 
during the educational preparation process concerning the alignment of: materials via creation 
and/or selection, learning plans, learner practice activities, and formative and summative 
assessment creation and design. This step is very important as it ensures the curriculum focuses 
on ability and not simply content. It is equally important to remember that good assessments, 
whether formative, summative, or standardized, measure ability of a content area. The TDCA 
process is performed using a common spreadsheet and a word processing program. 

A set of skill ability domain categories defining were chosen to be used in the columns 
consecutively following the first column. A skill ability domain is typically defined as a 
continuum that describes depth of ability. It is best to use a domain that is empirically expressed 
as steps of deeper learning or ability levels such as the commonly referenced Bloom‘s Taxonomy, 
with it‘s three different ability domains: cognitive, affective, and psychomotor (Gronlund & Linn, 
1990). Each domain has various stepped levels of ability depth. For example the cognitive 
domain has six levels of cognitive depth with the following descriptors: knowledge level, 
comprehension level, application level, analysis level, synthesis level, and evaluation level. 
However, a simpler post-positivistic taxonomy was sought that would logically represent 
learners ‘ depth of learning authentically while avoiding the tedious complexity of four or more 
levels. The traditional classifying categories for a task analysis use knowledge and skills 
(Jonassen et al., 1999). However, to stay aligned with a goal for this process to be authentically 
aligned with the CTE areas the categories need to represent cognitive, psychomotor, and affective 
as they exist in the real world, which is often very complex. The latter classifications seem to be 
too inclusive and limiting to the sensitivity of the depth of authentic ability. Another proposed 
categorical system of levels used four "targets” in a continuum and they are: knowledge, 
reasoning, skills, and product (Stiggins, Arter, Chappuis, & Chappuis, 2004; Chapuis & Chapuis, 
2002). However, the latter term "product” does not fit many authentic repair processes, such as 
those used in automotive service technology. Additionally, the category reasoning seems to fit 
some academic educational areas better than others so a second option, "concepts,” was merged. 

In reviewing various learning processes for CTE duty areas and tasks, two different 
ability domain categories emerged as possibilities for CTE taxonomy. Knowledge level and 
Concepts, Reasoning, and Problem Solving level remained prevalent for the first two categories 
of the two possible CTE ability domains for the low and middle ability levels, respectively. The 
third and highest category of thinking for the two ability domains include: Process Skills, 

Project, and Product either singularly or in combination. Therefore, the domain ability that 
included Problem Solving and Process Skills were adapted into the following categorical 
continuum to fit this CTE area of AST: Knowledge, Concepts / Reasoning, and Problem Solving 
and Process Skills. This classification continuum is simple, but more complex than a 
classification task analysis of simply knowledge and skills (Jonassen et al., 1999), and represents 
classifiable components of content knowledge and the cognitive processes needed to apply a set 
of authentic skills. 

The process was initiated using a spreadsheet with the ASTFS task descriptors listed as 
rows of the first column. The following three columns were labeled with our three chosen ability 
domain categories. Additionally, on the right of these columns we labeled three more columns: 
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Learning Plan & Materials Alignment 
Formative Assessment Alignment 
Summative Assessment Alignment 

These latter three columns would be used by the content educator to label and record 
educational component titles that reflect the objective learning processes and assessment. Table 5 
depicts the TDCA product table for the first sub-scale area of the BVRS with sample materials, 
learning plans, and learning practice examples inserted. The ability domain categories are the 
primary distinguishing differences of the TDCA product from a typical curriculum map 
(Curriculum Designers, 2006). 

The descriptive data is entered into the spreadsheet while simultaneously using the word 
processing software. The word processing software is set up to create a hierarchical outline 
listing of program of study, area of study, sub area, unit or duty, task, and objectives. Tables 6 
and 7 list a sample of the task domain listing of Learning Targets for knowledge, concepts, and 
process skills for the a sub-scale category of the BVRS in two different formats. Table 6 also 
lists the slightly adapted, but co mm on and stable hierarchical outline of the ASTFS task 
descriptors in [parentheses] (Bartel, 1976). It should be noted that the performance level or 
performance objective was considered equal to a learning target, but was omitted on purpose for 
the generalizability of the objective concept to allow a teacher to set the level in accordance with 
their facility and learning parameters. 
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Table 5. 

Sample Task Domain Categorization Analysis for an Area of the ASTFS 


Basic Vehicle 

Repair Skills Knowledge Concepts/ 

Sub-Scale Level Reasoning 

Categories 


Skills 


Learning 
Plan & 
Materials 
Alignment 


Formative Summative 
Assessment Assessment 
Alignment Alignment 


Oxy-Acetylene 
Torch Set-up 


Oxy-Acetylene 
Torch Storage 


Oxy-Acetylene Torch Safe Usage Scale 




Demonstrate 





torch set-up. 



Know and 

T 1‘nnPTctfi‘nn 

cylinder 

Text 

Kagan‘s 

identify torch 

LJ ilUd 5 fClllLl 
OYI/ilTITIfT 

exchange. 

chapter 3 

show down. 

components. 

UAlLllZlllg 

rVipmiral 

torch 

&4, 

Assignment 

related 

W'XXWXIXXW'CIX 

TP a Pfi PTI c 

operation. 

Oxy- 

correctives. 

terminology. 

ICclL/LlUllo, 

T T'nnPTctfi'nn 

torch 

Acetylene 

Laboratory 

OSHA 

LJ ilLld 5 fcxllU 
tVip rippH fni* 

shutdown. 

Torch Use 

assignment 

regulations. 

Lllk/ Xlk/V^Lt IvJl 

torch 

and Safety 

1 

and best 

proper lorcii 

relocating. 

Module, 

evaluation 

safety 

VdlVC 

and torch 

Lesson 

rubric 

practices 

sequencing 

temporary 

part one 

feedback 



storage 





procedures 




Understand 

the 

procedures Demonstrate 
that need to correct and 
be followed safe torch 
concerning temporary 
temporary and long- 
and long- term storage 
term storage practices 
in various 
conditions 


Text Kagan‘s 

chapter 3 show down, 
& 4, Assignment 
Oxy- correctives, 
Acetylene Laboratory 
Torch Use assignment 
and Safety 2 
Module, evaluation 
Lesson rubric 
part one feedback 


Laboratory 

assignment 

1 


completion 

of 

competency. 

Unit 

objective 

based 

assessment 

results 


Laboratory 

assignment 


2 


completion 

of 

competency. 

Unit 

objective 

based 

assessment 

results 


Oxy-Acetylene 
Torch Practices 


Know and 

identify torch Understand Demonstrate 
flame different torch 

characteristics types of welding and 
terminology torch flames cutting tip, 
and the and related pressure 
relationship uses, setting, 

between temperature cleaning, 

flame color, relationships, and usage 

shape, and procedures 

pressure, and limitations, 

flow 


Text Kagan‘s 

chapter 3 show down, 
& 4, Assignment 
Oxy- correctives. 
Acetylene Laboratory 
Torch Use assignment 
and Safety 3 
Module, evaluation 
Lesson rubric 

part two feedback 


Laboratory 

assignment 


3 


completion 

of 

competency. 

Unit 

objective 

based 

assessment 

result 
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Table 6, 


Automotive Service Technology Foundational Skills in First Format 

Sample Task Domain Listing of Objectives for Knowledge, Concepts, and Skills 

1. Automotive Service Technology Foundational Skills [Skill Area] 

A Basic Vehicle Repair Skills 

[Sub-Skill Area] 

1) Demonstrate Oxy-Acetylene Torch Use, Safety, and Maintenance [Unit or 

Duty] 

(1) Set-up Oxy- 

Acetylene Torches 
Safely [Skill or 
Task] 

(a) 1 will know and identify torch components, related 
terminology, OSHA regulations, and best safety practices. 

[Target] 

(b) 1 will demonstrate an understanding of oxidizing chemical 
reactions and proper torch valve sequencing. 

(c) 1 will demonstrate correct torch set-up, cylinder exchange, 
torch operation, torch shutdown, torch relocating, and torch 
temporary storage procedures. 

(2) Store Oxy-Acetylene Torches 

(a) 1 will demonstrate an understanding of the procedures that 
need to be followed concerning temporary and long-term 
storage in various conditions. 

(b) 1 will demonstrate correct and safe torch temporary and 
long-term storage practices. 

(3) Practice Safe Oxy-Acetylene Torch Usage 

(a) 1 will know and identify torch flame characteristics 
terminology and the relationship between flame color, shape, 
pressure, and flow. 

(b) 1 will demonstrate an understanding of the different types of 
torch flames, related uses, temperature relationships, and 
limitations. 

(c) 1 will demonstrate correct torch welding and cutting tip 
selection, pressure setting, lighting, cleaning, and usage 
procedures. 
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Table 7, 


Automotive Service Technology Foundational Skills in a Second Condensed Format 

Sample Task Domain Listing of Objectives for Knowledge, Concepts, and Skills 

1. Automotive Service Technology Foundational Skills 

LA. Basic Vehicle Repair Skills 

LA. 1. Demonstrate Oxy-Acetylene Torch Use, Safety, and Maintenance 

1 .A. 1 . 1 . Set-up Oxy-Acetylene Torches Safely 

LA. 1.1. a) 1 will know and identify torch components, related terminology, OSHA regulations, 

and best safety practices. 

1 .A. 1 . 1 .b) 1 will demonstrate an understanding of oxidizing chemical reactions and proper torch 

valve sequencing. 

LA. 1.1. c) 1 will demonstrate correct torch set-up, cylinder exchange, torch operation, torch 

shutdown, torch relocating, and torch temporary storage procedures. 

1 .A. 1 .2. Store Oxy-Acetylene Torches 

LA. 1.2. a) 1 will demonstrate an understanding of the procedures that need to be followed 

concerning temporary and long-term storage in various conditions. 

1 .A. 1 .2.b) 1 will demonstrate correct and safe torch temporary and long-term storage practices. 

1 .A. 1 .3. Practice Safe Oxy-Acetylene Torch Usage 

LA. 1.3. a) 1 will know and identify torch flame characteristics terminology and the relationship 

between flame color, shape, pressure, and flow. 

LA. 1.3. b) 1 will demonstrate an understanding of the different types of torch flames, related 

uses, temperature relationships, and limitations.. 

1.A.1.3.C) 1 will demonstrate correct torch welding and cutting tip selection, pressure setting, 

lighting, cleaning, and usage procedures. 


Table of Test Specifications Development 

A Table of Test Specifications (ToTS) provides a blue print for a proposed test 
construction plan. The ToTS can be used for formal formative or summative proficiency test 
construction planning. An ability domain is used in the assessment design process to ensure the 
test is sensitive to ability and that the test taker ‘s ability is measured and not simply the test 
taker ‘s knowledge of facts. The following section will describe the process used to design a 
ToTS for the ASTFSP Assessment. It is noted here that other domains could be used for different 
types of tests. The ASTFSP Assessment was designed to provide information about a person‘s 
ASTFS ability concerning cognitive processing in a quick and inexpensive manner and is 
therefore, a multiple choice paper and pencil assessment. Thus, the domain of choice was similar 
to that chosen for the TDCA process, but also references Bloom‘s Taxonomy for the cognitive 
processing domain more closely because a practical performance assessment was considered as 
being neither practical nor inexpensive. The three domains chosen are: Knowledge, 
Comprehension, and Application/Analysis levels. These three categories were chosen as they 
represent the cognitive ability that seemingly parallels those used for the TDCA. 
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The ToTS process starts by listing the skills in rows and the domain of skills across the 
columns. The row that follows the last ASTFS task in the sub-scale is labeled as a percentage 
and will be used to ensure a balance is obtained for each scale and sub-scale. Like-wise the very 
last row after the final sub-scale row is also labeled as percentage to ensure a balance in planned 
ability is maintained between each of the domain levels. The desired proportions were decided 
by several individual content experts after explaining the purpose of the assessment and 
reviewing the AMST results as well as the goal of the proportions. 

After the proportions are decided then the actual test item process will be tracked inside 
of the table. Inside of the table are proposed item numbers that match the domain as well as the 
skill for each category and sub-category. The goal of this latter process is to add the items to 
fiilfdl the ToTS proportion plan. Refer to Table 8 for a sample of the BVRS Table of Test 
Specifications. 


Table 8. 

Sample of the B VRS Scale of the ASTFSP Table of Test Specifications 


Basic Repair Skill Area Scale Validity Skill Levels 
Table 

Basic Vehicle Repair Skills Sub -Scale Knowledge 
Categories Level 

Comprehension Application 
Level Analysis Level 

/Percentage 

Oxy- Acetylene Torch Safe 
Scale 

Usage 




Oxy-Acetylene Torch Set-up 

21,25 

22 


12.50% 

Oxy- Acetylene Torch Storage 


26 


4.17% 

Oxy-Acetylene Torch Practices 


23 

24 

8.33% 

Sub-Scale Percentage of Test 
Percentage () 

Scale 8% (33%) 

13% (50%) 

4% (17%) 

26% 

Mechanical Aptitude & Safe Tool Use 




Scale 

Pneumatic tools and equipment 

27 

28 


8.33% 

Electrical power tools and equipment 29 


30 

8.33% 

Hand tool selection and use 

31 

32 


8.33% 

Mechanical Aptitude 



33,34,35,36 

16.67% 

Sub-Scale Percentage of Test 
Percentage () 

Scale 13% (27%) 

8% (18%) 

21% (45%) 

44% 

Facility Equipment Use and 
Scale 

Hoists and jack use 

Safety 

37 

38 

8.33% 

Fire extinguisher selection and use 39 



4.17% 

Ventilation 


40 


4.17% 

Personal Protective Equipment 

41,42 



8.33% 

Environmental Concerns 

43,44 



8.33% 

Sub-Scale Percentage of Test 
Percentage () 

Scale 21% (63%) 

8% (25%) 

4% (13%) 

30% 

Percentage of Test 

41.7% 

29.2% 

29.2% 

100.0% 
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Discussion, Implications, and Opportunities 
General Process Discussion 

The process described in this article details a qualitative analytical process for 
discovering and defining underlying, assumed, and prerequisite skills in the context area of 
Automotive Service Technology. Additionally, curricular and assessment alignment processes 
were described for the discovered ASTFS tasks. Context aside, this process could be replicated 
in almost any educational or career area such as machine trades, construction trades, automotive 
collision repair, heavy duty truck repair, welding, or other technical areas as the process 
encompasses a blend of knowledge, skills, and performance. It would make good sense to 
prepare students for defined and delineated specific level tasks or skills that are aligned with 
curriculum and internal assessments. High quality internal formative pre-test and post-test 
practices can guide the teaching and learning process using student feedback as a first stage 
intervention process. Additionally, a post test that are aligned with the skills of an existing 
summative assessments, such as state imposed proficiency tests, would allow second stage early 
interventions prior to actual high stakes testing. 


Process Discussion Specific to the Automotive World 

The defining, delineating, and development of both the ASTFS and the ASTFSP 
Assessment can assist schools, AST programs, and employers in evaluating the development 
level of AT‘s, prospective AT‘s, and AST students. AST students would likely benefit from 
effectively learning the aligned ASTFS skills most if they were to learn them prior to the NATEF 
task lists as they are underlying skill abilities all AT‘s would need. Additionally, the ASTFS 
skills would likely be transferable to many other transportation and industrial areas, making them 
ideal for career preparation skills. Interested organizations are invited to inquire or volunteer 
assistance with further research studies. 


Curricula and Assessment Alignment to Improve Adequate Yearly Progress 

Students that possess higher academic and technical skills have more career and college 
options and a higher probability for success (United States Department of Education, 2003). 
NCLB is focused on the accountability of graduates ‘ improvement skill levels for either post 
secondary education or the workforce. The Performance Measurement Initiative (PMI), 
mandated by the NCEB, encompasses the accountability of academic as well as Career and 
Technical Education (CTE) progress of students through assessment (United States Department 
of Education Office of Vocational and Adult Education, 2005). A CTE program that has aligned 
their curricula and internal assessments with both valid content and ability domains are likely to 
improve student learning progress each year as demonstrated on mandated external proficiency 
assessments. 
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